{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "77893d0a",
   "metadata": {},
   "source": [
    "# Agentic RAG with Llama Stack\n",
    "\n",
    "This notebook highlights how to integrate **Docling MCP** tools in the Agentic RAG available in Llama Stack.\n",
    "\n",
    "We will use the Llama Stack framework. To get an introduction to Llama Stack capabilities, including its current builtin tools, you can refer to the\n",
    "[Llama Stack Demos on OpenDataHub](https://github.com/opendatahub-io/llama-stack-demos) repository.\n",
    "\n",
    "This example will use the inline Milvus component available in the Llama Stack distributions.\n",
    "\n",
    "### Tools:\n",
    "\n",
    "We will use tools internal to Llama Stack and from the Docling MCP server that allow executing tasks such as:\n",
    "- [`mcp::docling`] converting a PDF file from a local or remote location into a unified document representation [DoclingDocument](https://docling-project.github.io/docling/concepts/docling_document/).\n",
    "- [`mcp::docling`] chunk and ingest the document in the Llama Stack vectordb.\n",
    "- [`builtin::rag/knowledge_search`] search in the document using agentic rag techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30845564",
   "metadata": {},
   "source": [
    "## Pre-Requisites\n",
    "\n",
    "Before starting this notebook, ensure that you have followed the instructions in the [README.md](./README.md) file to set up the following resources:\n",
    "- Inference model with Ollama.\n",
    "- Llama Stack server with the Ollama template [distribution-starter](https://hub.docker.com/r/llamastack/distribution-starter).\n",
    "\n",
    "  ⚠️ You need to enable the Milvus vector provider in Llama Stack with the additional environment variable `MILVUS_URL`:\n",
    "\n",
    "  ```shell\n",
    "  export LLAMA_STACK_PORT=8321\n",
    " \n",
    "  podman run \\\n",
    "    -it \\\n",
    "    --pull always \\\n",
    "    -p $LLAMA_STACK_PORT:$LLAMA_STACK_PORT \\\n",
    "    -v ~/.llama:/root/.llama \\\n",
    "    llamastack/distribution-starter \\\n",
    "    --port $LLAMA_STACK_PORT \\\n",
    "    --env OLLAMA_URL=http://host.containers.internal:11434 \\\n",
    "    --env MILVUS_URL=http://host.containers.internal:19530\n",
    "  ```\n",
    "\n",
    "- Docling MCP server.\n",
    "\n",
    "  ⚠️ You will need to run the Docling MCP server with the `conversion` and `llama-stack-rag` tools to complete the tasks in this notebook:\n",
    "\n",
    "  ```shell\n",
    "  uv run docling-mcp-server --transport streamable-http --port 8000 --host 0.0.0.0 conversion llama-stack-rag\n",
    "  ```\n",
    "\n",
    "You may want to create a virtual environment to run this notebook, for instance, with [uv](https://docs.astral.sh/uv/). Ensure to install the llama-stack optional dependencies, as well as the `examples` group dependencies:\n",
    "\n",
    "```bash\n",
    "uv venv\n",
    "source .venv/bin/activate\n",
    "uv sync --extra llama-stack --group examples\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c88519a5",
   "metadata": {},
   "source": [
    "\n",
    "## Setting Up this Notebook\n",
    "\n",
    "Rename or copy the [`.env.example`](./.env.examples) file to create a new file called `.env`. Most environmental variables are already set up with default values to run this notebook and they are aligned to the set up of the pre-requisites, like the Llama Stack server and the Docling MCP endpoints.\n",
    "\n",
    "```bash\n",
    "cp .env.example .env\n",
    "```\n",
    "\n",
    "### Environment variables required for this notebook\n",
    "\n",
    "- `BASE_URL`: the URL of the remote Llama Stack server. Defaults to `http://localhost:8321`.\n",
    "- `INFERENCE_MODEL`: the generative AI model id. Defaults to the Meta Llama 3.2 model (`meta-llama/Llama-3.2-3B-Instruct`).\n",
    "- `TEMPERATURE` (optional): the temperature to use during inference. Defaults to 0.0.\n",
    "- `TOP_P` (optional): the top_p parameter to use during inference. Defaults to 0.95.\n",
    "- `MAX_TOKENS` (optional): the maximum number of tokens that can be generated in the completion. Defaults to 4096.\n",
    "- `STREAM` (optional): set this to True to stream the output of the model/agent and False otherwise. Defaults to True.\n",
    "- `USE_PROMPT_CHAINING`: dictates if the prompt should be formatted as a few separate prompts to isolate each step or in a single turn.\n",
    "- `DOCLING_MCP_URL`: the URL for the Docling MCP server. If the client does not find the tool registered to the llama-stack instance, it will use this URL to register it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9ad7d58",
   "metadata": {},
   "source": [
    "## Necessary Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "07424c55",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">Settings</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">base_url</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">AnyHttpUrl</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">'http://localhost:8321/'</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">inference_model</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'ollama/llama3.2:3b-instruct-fp16'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">max_tokens</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4096</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">temperature</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.0</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">top_p</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0.95</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">stream</span>=<span style=\"color: #ff0000; text-decoration-color: #ff0000; font-style: italic\">False</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">use_prompt_chaining</span>=<span style=\"color: #00ff00; text-decoration-color: #00ff00; font-style: italic\">True</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">docling_mcp_url</span>=<span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">AnyHttpUrl</span><span style=\"font-weight: bold\">(</span><span style=\"color: #008000; text-decoration-color: #008000\">'http://host.containers.internal:8000/mcp'</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">vdb_provider</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'milvus'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">vdb_embedding</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'all-MiniLM-L6-v2'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #808000; text-decoration-color: #808000\">vdb_embedding_dimension</span>=<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">384</span>\n",
       "<span style=\"font-weight: bold\">)</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;35mSettings\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mbase_url\u001b[0m=\u001b[1;35mAnyHttpUrl\u001b[0m\u001b[1m(\u001b[0m\u001b[32m'http://localhost:8321/'\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33minference_model\u001b[0m=\u001b[32m'ollama/llama3.2:3b-instruct-fp16'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mmax_tokens\u001b[0m=\u001b[1;36m4096\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mtemperature\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.0\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mtop_p\u001b[0m=\u001b[1;36m0\u001b[0m\u001b[1;36m.95\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mstream\u001b[0m=\u001b[3;91mFalse\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33muse_prompt_chaining\u001b[0m=\u001b[3;92mTrue\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mdocling_mcp_url\u001b[0m=\u001b[1;35mAnyHttpUrl\u001b[0m\u001b[1m(\u001b[0m\u001b[32m'http://host.containers.internal:8000/mcp'\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mvdb_provider\u001b[0m=\u001b[32m'milvus'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mvdb_embedding\u001b[0m=\u001b[32m'all-MiniLM-L6-v2'\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[33mvdb_embedding_dimension\u001b[0m=\u001b[1;36m384\u001b[0m\n",
       "\u001b[1m)\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import logging\n",
    "import uuid\n",
    "\n",
    "from llama_stack_client import Agent\n",
    "from llama_stack_client import LlamaStackClient\n",
    "from llama_stack_client.lib.agents.event_logger import EventLogger\n",
    "from pydantic import NonNegativeFloat, AnyHttpUrl\n",
    "from pydantic_settings import BaseSettings, SettingsConfigDict\n",
    "from rich.pretty import pprint\n",
    "\n",
    "from src.utils import step_printer, user_printer\n",
    "\n",
    "# set the logger\n",
    "logger = logging.getLogger(__name__)\n",
    "if not logger.hasHandlers():\n",
    "    logger.setLevel(logging.INFO)\n",
    "    stream_handler = logging.StreamHandler()\n",
    "    stream_handler.setLevel(logging.INFO)\n",
    "    formatter = logging.Formatter(\"%(message)s\")\n",
    "    stream_handler.setFormatter(formatter)\n",
    "    logger.addHandler(stream_handler)\n",
    "\n",
    "\n",
    "# access the environment variables\n",
    "class Settings(BaseSettings):\n",
    "    base_url: AnyHttpUrl = \"http://localhost:8321\"\n",
    "    inference_model: str = \"ollama/llama3.2:3b-instruct-fp16\"\n",
    "    max_tokens: int = 4096\n",
    "    temperature: NonNegativeFloat = 0.0\n",
    "    top_p: float = 0.95\n",
    "    stream: bool = False\n",
    "    use_prompt_chaining: bool = True\n",
    "\n",
    "    docling_mcp_url: AnyHttpUrl = \"http://host.containers.internal:8000/mcp\"\n",
    "\n",
    "    vdb_provider: str = \"milvus\"\n",
    "    vdb_embedding: str = \"all-MiniLM-L6-v2\"\n",
    "    vdb_embedding_dimension: int = 384\n",
    "    # vdb_embedding_window: int = 256\n",
    "\n",
    "    model_config = SettingsConfigDict(\n",
    "        env_file=\".env\", env_file_encoding=\"utf-8\", extra=\"ignore\"\n",
    "    )\n",
    "\n",
    "\n",
    "settings = Settings()\n",
    "pprint(settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ad79b36",
   "metadata": {},
   "source": [
    "## Setting Up the Server Connection\n",
    "\n",
    "Establish the connection to your Llama Stack server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f698f34c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connected to Llama Stack server @ http://localhost:8321/\n"
     ]
    }
   ],
   "source": [
    "client = LlamaStackClient(base_url=str(settings.base_url))\n",
    "print(f\"Connected to Llama Stack server @ {client.base_url}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed93b8e9",
   "metadata": {},
   "source": [
    "## Initializing the Inference Parameters\n",
    "\n",
    "Fetch the inference-related parameters from the corresponding environment variables and convert them to the format Llama Stack expects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a2dee538",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inference Parameters:\n",
      "\tSampling Parameters: {'strategy': {'type': 'greedy'}, 'max_tokens': 4096}\n",
      "\tstream: False\n"
     ]
    }
   ],
   "source": [
    "if settings.temperature > 0.0:\n",
    "    strategy = {\n",
    "        \"type\": \"top_p\",\n",
    "        \"temperature\": settings.temperature,\n",
    "        \"top_p\": settings.top_p,\n",
    "    }\n",
    "else:\n",
    "    strategy = {\"type\": \"greedy\"}\n",
    "\n",
    "# sampling_params will later be used to pass the parameters to Llama Stack Agents/Inference APIs\n",
    "sampling_params = {\n",
    "    \"strategy\": strategy,\n",
    "    \"max_tokens\": settings.max_tokens,\n",
    "}\n",
    "\n",
    "print(\n",
    "    f\"Inference Parameters:\\n\\tSampling Parameters: {sampling_params}\\n\\tstream: {settings.stream}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6eba1f1",
   "metadata": {},
   "source": [
    "## Validate that the Docling MCP tools are available in the Llama Stack instance\n",
    "\n",
    "When an instance of Llama Stack is redeployed, it may be the case that the tools will need to be re-registered. Also if a tool is already registered with a Llama Stack instance, trying to register another one with the same `toolgroup_id` will throw you an error.\n",
    "\n",
    "For this reason, it is recommended to validate your tools and toolgroups. The following code will check that `mcp::docling` tools are correctly registered, and if not it will attempt to register them using their specific endpoints."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2fa74729",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:httpx:HTTP Request: GET http://localhost:8321/v1/tools \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: GET http://localhost:8321/v1/tools \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Your Llama Stack server is registered with the following tool groups @ {'builtin::rag', 'builtin::websearch', 'mcp::docling'} \n",
      "\n"
     ]
    }
   ],
   "source": [
    "registered_tools = client.tools.list()\n",
    "registered_toolgroups = [t.toolgroup_id for t in registered_tools]\n",
    "if \"mcp::docling\" not in registered_toolgroups:\n",
    "    client.toolgroups.register(\n",
    "        toolgroup_id=\"mcp::docling\",\n",
    "        provider_id=\"model-context-protocol\",\n",
    "        mcp_endpoint={\"uri\": str(settings.docling_mcp_url)},\n",
    "    )\n",
    "\n",
    "registered_tools = client.tools.list()\n",
    "registered_toolgroups = [t.toolgroup_id for t in registered_tools]\n",
    "print(\n",
    "    f\"Your Llama Stack server is registered with the following tool groups @ {set(registered_toolgroups)} \\n\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bab507be",
   "metadata": {},
   "source": [
    "## Defining our Agent - Prompt Chaining"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aec5c8c",
   "metadata": {},
   "source": [
    "We define an agent provided with the **Docling MCP** tools together with the built-in knowledge_search. The agent should be able to accomplish the following tasks in a multi-step, multi-tool approach:\n",
    "\n",
    "1. Converting a PDF file into the `DoclingDocument` format.\n",
    "2. Ingest the results in the vector db.\n",
    "3. Search in the vector db using an agentic/multi-step approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b3a8e40d",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_prompt = \"\"\"You are a helpful assistant. You have access to a number of tools.\n",
    "Whenever a tool is called, be sure to return the Response in a friendly and helpful tone.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "39e0e316",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:httpx:HTTP Request: POST http://localhost:8321/v1/vector-dbs \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST http://localhost:8321/v1/agents \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: GET http://localhost:8321/v1/tools?toolgroup_id=mcp%3A%3Adocling%2Fconvert_document_into_docling_document \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: GET http://localhost:8321/v1/tools?toolgroup_id=mcp%3A%3Adocling%2Finsert_document_to_vectordb \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: GET http://localhost:8321/v1/tools?toolgroup_id=builtin%3A%3Arag%2Fknowledge_search \"HTTP/1.1 200 OK\"\n"
     ]
    }
   ],
   "source": [
    "# define the name of the vectordb collection to use\n",
    "vector_db_id = f\"test_vector_db_{uuid.uuid4()}\"\n",
    "\n",
    "# define and register the document collection to be used\n",
    "client.vector_dbs.register(\n",
    "    vector_db_id=vector_db_id,\n",
    "    embedding_model=settings.vdb_embedding,\n",
    "    embedding_dimension=settings.vdb_embedding_dimension,\n",
    "    provider_id=settings.vdb_provider,\n",
    ")\n",
    "\n",
    "\n",
    "# Create simple agent with tools\n",
    "agent = Agent(\n",
    "    client=client,\n",
    "    model=settings.inference_model,  # replace this with model_id to get the value of INFERENCE_MODEL_ID environment variable\n",
    "    instructions=model_prompt,  # update system prompt based on the model you are using\n",
    "    tools=[\n",
    "        dict(\n",
    "            name=\"mcp::docling/convert_document_into_docling_document\",\n",
    "            args={},\n",
    "        ),\n",
    "        dict(\n",
    "            name=\"mcp::docling/insert_document_to_vectordb\",\n",
    "            args={\n",
    "                \"vector_db_id\": vector_db_id,\n",
    "            },\n",
    "        ),\n",
    "        dict(\n",
    "            name=\"builtin::rag/knowledge_search\",\n",
    "            args={\n",
    "                \"vector_db_ids\": [\n",
    "                    vector_db_id\n",
    "                ],  # list of IDs of document collections to consider during retrieval\n",
    "            },\n",
    "        ),\n",
    "    ],\n",
    "    tool_config={\"tool_choice\": \"auto\"},\n",
    "    sampling_params=sampling_params,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a69aab5c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:httpx:HTTP Request: POST http://localhost:8321/v1/agents/8d7da0f4-b7dd-4a28-b6b5-961912e757fb/session \"HTTP/1.1 200 OK\"\n",
      "INFO:httpx:HTTP Request: POST http://localhost:8321/v1/agents/8d7da0f4-b7dd-4a28-b6b5-961912e757fb/session/5c2ca9e6-f507-4b16-a2d4-94d4c3b670ed/turn \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "👤 User Query:\n",
      "\u001b[36mConvert the PDF document on https://arxiv.org/pdf/2206.01062 to DoclingDocument.\u001b[0m\n",
      "\n",
      "---------- 📍 Step 1: InferenceStep ----------\n",
      "🛠️ Tool call Generated:\n",
      "\u001b[35mTool call: convert_document_into_docling_document, Arguments: {'source': 'https://arxiv.org/pdf/2206.01062'}\u001b[0m\n",
      "\n",
      "---------- 📍 Step 2: ToolExecutionStep ----------\n",
      "🔧 Executing tool...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'{\\n  \"from_cache\": false,\\n  \"document_key\": \"868f49ae1f0e66e82238a8aea43fd30b\"\\n}'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\n  \"from_cache\": false,\\n  \"document_key\": \"868f49ae1f0e66e82238a8aea43fd30b\"\\n\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:httpx:HTTP Request: POST http://localhost:8321/v1/agents/8d7da0f4-b7dd-4a28-b6b5-961912e757fb/session/5c2ca9e6-f507-4b16-a2d4-94d4c3b670ed/turn \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---------- 📍 Step 3: InferenceStep ----------\n",
      "🤖 Model Response:\n",
      "\u001b[35mThe conversion of the PDF document from https://arxiv.org/pdf/2206.01062 to DoclingDocument was successful. The resulting document has a unique cache key '868f49ae1f0e66e82238a8aea43fd30b'. If you want to insert this document into a vectordb, you can use the insert_document_to_vectordb tool with the following parameters:\n",
      "\n",
      "[insert_document_to_vectordb(document_key='868f49ae1f0e66e82238a8aea43fd30b', vector_db_id='your_vector_db_id')]\n",
      "\u001b[0m\n",
      "========== Query processing completed ========== \n",
      "\n",
      "👤 User Query:\n",
      "\u001b[36mIngest the document.\u001b[0m\n",
      "\n",
      "---------- 📍 Step 1: InferenceStep ----------\n",
      "🛠️ Tool call Generated:\n",
      "\u001b[35mTool call: insert_document_to_vectordb, Arguments: {'document_key': '868f49ae1f0e66e82238a8aea43fd30b', 'vector_db_id': 'your_vector_db_id'}\u001b[0m\n",
      "\n",
      "---------- 📍 Step 2: ToolExecutionStep ----------\n",
      "🔧 Executing tool...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'{\\n  \"vector_db_id\": \"test_vector_db_d9755fa0-71be-4b0d-af5c-46d9cf4f5741\"\\n}'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m'\u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\n  \"vector_db_id\": \"test_vector_db_d9755fa0-71be-4b0d-af5c-46d9cf4f5741\"\\n\u001b[0m\u001b[32m}\u001b[0m\u001b[32m'\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:httpx:HTTP Request: POST http://localhost:8321/v1/agents/8d7da0f4-b7dd-4a28-b6b5-961912e757fb/session/5c2ca9e6-f507-4b16-a2d4-94d4c3b670ed/turn \"HTTP/1.1 200 OK\"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---------- 📍 Step 3: InferenceStep ----------\n",
      "🤖 Model Response:\n",
      "\u001b[35mThe document has been ingested into the vectordb. You can now use the vector_db_id 'test_vector_db_d9755fa0-71be-4b0d-af5c-46d9cf4f5741' for knowledge searches. \n",
      "\n",
      "You can try searching for information using the knowledge_search tool with this vector_db_id:\n",
      "\n",
      "[knowledge_search(query=\"your_search_query\")]\n",
      "\u001b[0m\n",
      "========== Query processing completed ========== \n",
      "\n",
      "👤 User Query:\n",
      "\u001b[36mAnswer with the document knowledge in the vectordb: How many pages were manually annotated in the dataset?\u001b[0m\n",
      "\n",
      "---------- 📍 Step 1: InferenceStep ----------\n",
      "🛠️ Tool call Generated:\n",
      "\u001b[35mTool call: knowledge_search, Arguments: {'query': 'number of pages manually annotated in dataset'}\u001b[0m\n",
      "\n",
      "---------- 📍 Step 2: ToolExecutionStep ----------\n",
      "🔧 Executing tool...\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'knowledge_search tool found 5 chunks:\\nBEGIN of knowledge_search tool results.\\n'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Result 1\\nContent: 3 THE DOCLAYNET DATASET\\nDocLayNet contains 80863 PDF pages. Among these, 7059 carry two instances of human annotations, and 1591 carry three. This amounts to 91104 total annotation instances. The annotations provide layout information in the shape of labeled, rectangular boundingboxes. We define 11 distinct labels for layout features, namely Caption , Footnote , Formula , List-item , Page-footer , Page-header , Picture , Section-header , Table , Text , and Title . Our reasoning for picking this particular label set is detailed in Section 4.\\nIn addition to open intellectual property constraints for the source documents, we required that the documents in DocLayNet adhere to a few conditions. Firstly, we kept scanned documents\\nFigure 2: Distribution of DocLayNet pages across document categories.\\nMetadata: {'chunk_id': '7156212269791437020-00012', 'document_id': '7156212269791437020', 'source': None, 'doc_items': ['#/texts/339', '#/texts/340', '#/texts/343']}\\n\"</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Result 2\\nContent: 1 INTRODUCTION\\n- (1) Human Annotation : In contrast to PubLayNet and DocBank, we relied on human annotation instead of automation approaches to generate the data set.\\n- (2) Large Layout Variability : We include diverse and complex layouts from a large variety of public sources.\\n- (3) Detailed Label Set : We define 11 class labels to distinguish layout features in high detail. PubLayNet provides 5 labels; DocBank provides 13, although not a superset of ours.\\n- (4) Redundant Annotations : A fraction of the pages in the DocLayNet data set carry more than one human annotation.\\n1 https://developer.ibm.com/exchanges/data/all/doclaynet\\nThis enables experimentation with annotation uncertainty and quality control analysis.\\n- (5) Pre-defined Train-, Test- &amp; Validation-set : Like DocBank, we provide fixed train-, test- &amp; validation-sets to ensure proportional representation of the class-labels. Further, we prevent leakage of unique layouts across sets, which has a large effect on model accuracy scores.\\nMetadata: {'chunk_id': '7156212269791437020-00009', 'document_id': '7156212269791437020', 'source': None, 'doc_items': ['#/texts/326', '#/texts/327', '#/texts/328', '#/texts/329', '#/texts/330', '#/texts/331', '#/texts/332']}\\n\"</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Result 3\\nContent: 4 ANNOTATION CAMPAIGN\\nPreparation work included uploading and parsing the sourced PDF documents in the Corpus Conversion Service (CCS) [22], a cloud-native platform which provides a visual annotation interface and allows for dataset inspection and analysis. The annotation interface of CCS is shown in Figure 3. The desired balance of pages between the different document categories was achieved by selective subsampling of pages with certain desired properties. For example, we made sure to include the title page of each document and bias the remaining page selection to those with figures or tables. The latter was achieved by leveraging pre-trained object detection models from PubLayNet, which helped us estimate how many figures and tables a given page contains.\\nMetadata: {'chunk_id': '7156212269791437020-00033', 'document_id': '7156212269791437020', 'source': None, 'doc_items': ['#/texts/394']}\\n\"</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'Result 4\\nContent: 3 THE DOCLAYNET DATASET\\nto a minimum, since they introduce difficulties in annotation (see Section 4). As a second condition, we focussed on medium to large documents ( &gt; 10 pages) with technical content, dense in complex tables, figures, plots and captions. Such documents carry a lot of information value, but are often hard to analyse with high accuracy due to their challenging layouts. Counterexamples of documents not included in the dataset are receipts, invoices, hand-written documents or photographs showing \\'text in the wild\".\\nMetadata: {\\'chunk_id\\': \\'7156212269791437020-00013\\', \\'document_id\\': \\'7156212269791437020\\', \\'source\\': None, \\'doc_items\\': [\\'#/texts/356\\']}\\n'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">\"Result 5\\nContent: 4 ANNOTATION CAMPAIGN\\nThe annotation campaign was carried out in four phases. In phase one, we identified and prepared the data sources for annotation. In phase two, we determined the class labels and how annotations should be done on the documents in order to obtain maximum consistency. The latter was guided by a detailed requirement analysis and exhaustive experiments. In phase three, we trained the annotation staff and performed exams for quality assurance. In phase four,\\n\\nTable 1: DocLayNet dataset overview. Along with the frequency of each class label, we present the relative occurrence (as % of row 'Total') in the train, test and validation sets. The inter-annotator agreement is computed as the mAP@0.5-0.95 metric between pairwise annotations from the triple-annotated pages, from which we obtain accuracy ranges.\\nMetadata: {'chunk_id': '7156212269791437020-00019', 'document_id': '7156212269791437020', 'source': None, 'doc_items': ['#/texts/365', '#/texts/367', '#/tables/0']}\\n\"</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'END of knowledge_search tool results.\\n'</span>, <span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span><span style=\"font-weight: bold\">)</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">TextContentItem</span><span style=\"font-weight: bold\">(</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">text</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'The above results were retrieved to help answer the user\\'s query: \"number of pages manually annotated in dataset\". Use them as supporting information only in answering this query.\\n'</span>,\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   │   </span><span style=\"color: #808000; text-decoration-color: #808000\">type</span>=<span style=\"color: #008000; text-decoration-color: #008000\">'text'</span>\n",
       "<span style=\"color: #7fbf7f; text-decoration-color: #7fbf7f\">│   </span><span style=\"font-weight: bold\">)</span>\n",
       "<span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m'knowledge_search tool found 5 chunks:\\nBEGIN of knowledge_search tool results.\\n'\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m\"Result\u001b[0m\u001b[32m 1\\nContent: 3 THE DOCLAYNET DATASET\\nDocLayNet contains 80863 PDF pages. Among these, 7059 carry two instances of human annotations, and 1591 carry three. This amounts to 91104 total annotation instances. The annotations provide layout information in the shape of labeled, rectangular boundingboxes. We define 11 distinct labels for layout features, namely Caption , Footnote , Formula , List-item , Page-footer , Page-header , Picture , Section-header , Table , Text , and Title . Our reasoning for picking this particular label set is detailed in Section 4.\\nIn addition to open intellectual property constraints for the source documents, we required that the documents in DocLayNet adhere to a few conditions. Firstly, we kept scanned documents\\nFigure 2: Distribution of DocLayNet pages across document categories.\\nMetadata: \u001b[0m\u001b[32m{\u001b[0m\u001b[32m'chunk_id': '7156212269791437020-00012', 'document_id': '7156212269791437020', 'source': None, 'doc_items': \u001b[0m\u001b[32m[\u001b[0m\u001b[32m'#/texts/339', '#/texts/340', '#/texts/343'\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\\n\"\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m\"Result\u001b[0m\u001b[32m 2\\nContent: 1 INTRODUCTION\\n- \u001b[0m\u001b[32m(\u001b[0m\u001b[32m1\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Human Annotation : In contrast to PubLayNet and DocBank, we relied on human annotation instead of automation approaches to generate the data set.\\n- \u001b[0m\u001b[32m(\u001b[0m\u001b[32m2\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Large Layout Variability : We include diverse and complex layouts from a large variety of public sources.\\n- \u001b[0m\u001b[32m(\u001b[0m\u001b[32m3\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Detailed Label Set : We define 11 class labels to distinguish layout features in high detail. PubLayNet provides 5 labels; DocBank provides 13, although not a superset of ours.\\n- \u001b[0m\u001b[32m(\u001b[0m\u001b[32m4\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Redundant Annotations : A fraction of the pages in the DocLayNet data set carry more than one human annotation.\\n1 https://developer.ibm.com/exchanges/data/all/doclaynet\\nThis enables experimentation with annotation uncertainty and quality control analysis.\\n- \u001b[0m\u001b[32m(\u001b[0m\u001b[32m5\u001b[0m\u001b[32m)\u001b[0m\u001b[32m Pre-defined Train-, Test- & Validation-set : Like DocBank, we provide fixed train-, test- & validation-sets to ensure proportional representation of the class-labels. Further, we prevent leakage of unique layouts across sets, which has a large effect on model accuracy scores.\\nMetadata: \u001b[0m\u001b[32m{\u001b[0m\u001b[32m'chunk_id': '7156212269791437020-00009', 'document_id': '7156212269791437020', 'source': None, 'doc_items': \u001b[0m\u001b[32m[\u001b[0m\u001b[32m'#/texts/326', '#/texts/327', '#/texts/328', '#/texts/329', '#/texts/330', '#/texts/331', '#/texts/332'\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\\n\"\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m\"Result\u001b[0m\u001b[32m 3\\nContent: 4 ANNOTATION CAMPAIGN\\nPreparation work included uploading and parsing the sourced PDF documents in the Corpus Conversion Service \u001b[0m\u001b[32m(\u001b[0m\u001b[32mCCS\u001b[0m\u001b[32m)\u001b[0m\u001b[32m \u001b[0m\u001b[32m[\u001b[0m\u001b[32m22\u001b[0m\u001b[32m]\u001b[0m\u001b[32m, a cloud-native platform which provides a visual annotation interface and allows for dataset inspection and analysis. The annotation interface of CCS is shown in Figure 3. The desired balance of pages between the different document categories was achieved by selective subsampling of pages with certain desired properties. For example, we made sure to include the title page of each document and bias the remaining page selection to those with figures or tables. The latter was achieved by leveraging pre-trained object detection models from PubLayNet, which helped us estimate how many figures and tables a given page contains.\\nMetadata: \u001b[0m\u001b[32m{\u001b[0m\u001b[32m'chunk_id': '7156212269791437020-00033', 'document_id': '7156212269791437020', 'source': None, 'doc_items': \u001b[0m\u001b[32m[\u001b[0m\u001b[32m'#/texts/394'\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\\n\"\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m'Result 4\\nContent: 3 THE DOCLAYNET DATASET\\nto a minimum, since they introduce difficulties in annotation \u001b[0m\u001b[32m(\u001b[0m\u001b[32msee Section 4\u001b[0m\u001b[32m)\u001b[0m\u001b[32m. As a second condition, we focussed on medium to large documents \u001b[0m\u001b[32m(\u001b[0m\u001b[32m > 10 pages\u001b[0m\u001b[32m)\u001b[0m\u001b[32m with technical content, dense in complex tables, figures, plots and captions. Such documents carry a lot of information value, but are often hard to analyse with high accuracy due to their challenging layouts. Counterexamples of documents not included in the dataset are receipts, invoices, hand-written documents or photographs showing \\'text in the wild\".\\nMetadata: \u001b[0m\u001b[32m{\u001b[0m\u001b[32m\\'chunk_id\\': \\'7156212269791437020-00013\\', \\'document_id\\': \\'7156212269791437020\\', \\'source\\': None, \\'doc_items\\': \u001b[0m\u001b[32m[\u001b[0m\u001b[32m\\'#/texts/356\\'\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\\n'\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m\"Result\u001b[0m\u001b[32m 5\\nContent: 4 ANNOTATION CAMPAIGN\\nThe annotation campaign was carried out in four phases. In phase one, we identified and prepared the data sources for annotation. In phase two, we determined the class labels and how annotations should be done on the documents in order to obtain maximum consistency. The latter was guided by a detailed requirement analysis and exhaustive experiments. In phase three, we trained the annotation staff and performed exams for quality assurance. In phase four,\\n\\nTable 1: DocLayNet dataset overview. Along with the frequency of each class label, we present the relative occurrence \u001b[0m\u001b[32m(\u001b[0m\u001b[32mas % of row 'Total'\u001b[0m\u001b[32m)\u001b[0m\u001b[32m in the train, test and validation sets. The inter-annotator agreement is computed as the mAP@0.5-0.95 metric between pairwise annotations from the triple-annotated pages, from which we obtain accuracy ranges.\\nMetadata: \u001b[0m\u001b[32m{\u001b[0m\u001b[32m'chunk_id': '7156212269791437020-00019', 'document_id': '7156212269791437020', 'source': None, 'doc_items': \u001b[0m\u001b[32m[\u001b[0m\u001b[32m'#/texts/365', '#/texts/367', '#/tables/0'\u001b[0m\u001b[32m]\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\\n\"\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m'END of knowledge_search tool results.\\n'\u001b[0m, \u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\u001b[1m)\u001b[0m,\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1;35mTextContentItem\u001b[0m\u001b[1m(\u001b[0m\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtext\u001b[0m=\u001b[32m'The above results were retrieved to help answer the user\\'s query: \"number of pages manually annotated in dataset\". Use them as supporting information only in answering this query.\\n'\u001b[0m,\n",
       "\u001b[2;32m│   │   \u001b[0m\u001b[33mtype\u001b[0m=\u001b[32m'text'\u001b[0m\n",
       "\u001b[2;32m│   \u001b[0m\u001b[1m)\u001b[0m\n",
       "\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---------- 📍 Step 3: InferenceStep ----------\n",
      "🤖 Model Response:\n",
      "\u001b[35mAccording to the search results, 91104 annotation instances were created for the DocLayNet dataset. This corresponds to a total of 7059 pages with two annotations and 1591 pages with three annotations. Therefore, the answer is:\n",
      "\n",
      "There are 91104 annotation instances in the DocLayNet dataset, which corresponds to 7059 pages with two annotations and 1591 pages with three annotations.\n",
      "\u001b[0m\n",
      "========== Query processing completed ========== \n",
      "\n"
     ]
    }
   ],
   "source": [
    "user_prompts = [\n",
    "    \"Convert the PDF document on https://arxiv.org/pdf/2206.01062 to DoclingDocument.\",\n",
    "    \"Ingest the document.\",\n",
    "    \"Answer with the document knowledge in the vectordb: How many pages were manually annotated in the dataset?\",\n",
    "]\n",
    "session_id = agent.create_session(f\"docling-session_{uuid.uuid4()}\")\n",
    "\n",
    "for i, prompt in enumerate(user_prompts):\n",
    "    user_printer(prompt)\n",
    "    response = agent.create_turn(\n",
    "        messages=[{\"role\": \"user\", \"content\": prompt}],\n",
    "        session_id=session_id,\n",
    "        stream=settings.stream,\n",
    "    )\n",
    "    if settings.stream:\n",
    "        for log in EventLogger().log(response):\n",
    "            log.print()\n",
    "    else:\n",
    "        step_printer(\n",
    "            response.steps\n",
    "        )  # print the steps of an agent's response in a formatted way."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
